Skip to content

fix(EPv2): add a tuned flydsl EP8 table for MI300X - #795

Open
jeffdaily wants to merge 2 commits into
ROCm:mainfrom
jeffdaily:tune/mi300x-flydsl-ep8
Open

jeffdaily wants to merge 2 commits into
ROCm:mainfrom
jeffdaily:tune/mi300x-flydsl-ep8

Conversation

@jeffdaily

@jeffdaily jeffdaily commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

On MI300X the flydsl EP v2 backend has no tuned geometry, so a default-constructed op fails (#794). lookup() falls back to _cu_scaled_default(), which sets combine_block_num to the CU count (304), and construction asserts combine block_num 304 exceeds xdb_flag_slots 256. MI300X HF parts (PCI DID 0x74a9) were also missing from _DID_TO_MODEL, so they never reached the MI300X entry.

Part of #794: this fixes MI300X. The issue stays open for the clamp, since any other untuned part with more than 256 CUs still hits the assert.

Technical Details

  • utils.py: map DID 0x74a9 to mi300x. Only the flydsl v2 lookup changes behaviour; the HIP v2 and internode tables have no mi300x key, and the v1 path already resolved this part to mi300x by device name.
  • tuning_configs.py: a measured EP8 schedule for MI300X, used for bf16 and fp8 and for hidden 7168, 4096 and 2048 (4096 and 2048 reuse the 7168 schedule, as the MI325X table does). Every block count is 128 or less.
tok/rank dispatch combine
<= 128 128x8 64x8
<= 1024 128x8 32x8
> 1024 128x16 32x8

The MI325X schedule is not aliased: its combine geometry is 13-19 % slower here from 256 tokens up.

Measured on 8x MI300X HF over xGMI with bench_ep.py (flydsl, pull combine, graph mode, hidden 7168, topk 8, EPR 32, tokens 8 to 8192, bf16 and fp8 dispatch). Dispatch block {64..304} x warp {4,8,16} and combine block {32..256} x warp {2,4,8,16} were swept per leg, the minimum mean latency was taken per bucket, and the winners and runners-up were re-run twice (spread under 1 % from 128 tokens up). bf16 and fp8 pick the same geometry.

EP8 bf16 dispatch+combine, us, graph mode, mean of 3 rounds:

tok/rank this PR MI325X schedule CU fallback with combine capped at 256
128 94 101 108
1024 556 584 788
4096 2048 2354 2975
8192 4039 4642 6020

So a clamp alone would leave MI300X about a third slower at large token counts. Below 128 tokens every geometry measured was within run-to-run noise.

Not covered: EP2 and EP4 have no table entry and get the single-shot default (dispatch 128x16, combine 32x8), which passes but is untuned. Only the 0x74a9 part was measured; 0x74a1 shares the table. The runs used flydsl 0.4.0 (an unpinned install today), while docker/Dockerfile.dev pins flydsl<0.3.

A second commit maps the other gfx942 device IDs from libdrm's amdgpu.ids that had no model: 0x74b5 (MI300X VF) and 0x74bd (MI300X HF) to mi300x, 0x74b9 (MI325X VF) to mi325x, and 0x74a8 (MI308X HF) to mi308x. Without them, MI300X-class parts with these IDs hit the same assert, and MI308X HF gets an untuned default. gfx950 VFs already reach the MI355X table through the arch fallback. Checked with detect_model() against a stubbed topology(); not run on VF hardware.

Test Plan

8x MI300X (gfx942), flydsl default backend, no geometry overrides. On main every one of these fails at op construction.

torchrun --standalone --nproc_per_node={2,4,8} test_op.py          # also TUNED=1, DTYPE=fp8 TUNED=1, HIDDEN=7168 SWEEP=8,128,512,2048
torchrun --standalone --nproc_per_node={2,4,8} test_ep_backend_parity.py   # also HIDDEN=7168 TOPK=8 EPR=32
torchrun --standalone --nproc_per_node=8 test_graph_capture.py     # also HIDDEN=7168 CT=4096
REPLAY=1 torchrun --standalone --nproc_per_node=8 test_op.py
RAGGED=1 SWEEP=8,128,512,2048 torchrun --standalone --nproc_per_node=8 test_op.py
pytest tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_intranode.py   # 17 passed, 1 skipped (fp4, gfx950-only)

All passed. black and ruff are clean on the changed lines.

This PR was authored with Claude (AI assistant) and reviewed by me.

On MI300X (gfx942, 304 CU) the flydsl EP v2 backend had no tuned geometry. lookup() fell back to _cu_scaled_default(), which sets combine_block_num to the CU count (304), and op construction asserted `combine block_num 304 exceeds xdb_flag_slots 256`. MI300X HF parts (PCI DID 0x74a9) were not in _DID_TO_MODEL either, so they never reached the MI300X entry at all.

Part of ROCm#794 (MI300X only; other untuned parts above 256 CUs still need the clamp). This adds 0x74a9 to _DID_TO_MODEL and a measured MI300X schedule: dispatch 128x8 (128x16 above 1024 tokens), combine 64x8 up to 128 tokens and 32x8 above. Every block_num is <= 128, well under both the CU count and xdb_flag_slots. The MI325X schedule was not aliased, because its combine geometry is 13-19 % slower here at >= 256 tokens.

How it was measured: 8x MI300X HF over xGMI, bench_ep.py with BACKENDS=flydsl COMB_MODE=pull MODES=graph, hidden 7168, topk 8, EPR 32, tokens 8..8192, bf16 and fp8 dispatch. Dispatch block {64..304} x warp {4,8,16} and combine block {32..256} x warp {2,4,8,16} were swept per leg. The minimum mean latency was picked per bucket, and the winners and runners-up were re-run twice (spread < 1 % from 128 tokens up). bf16 and fp8 pick the same geometry, so one schedule serves both.

EP8 bf16 dispatch+combine pair, us, graph mode, mean of 3 rounds:

| tok/rank | this table | MI325X schedule | CU fallback, combine capped 256 |
|---|---|---|---|
| 128 | 94 | 101 | 108 |
| 1024 | 556 | 584 | 788 |
| 4096 | 2048 | 2354 | 2975 |
| 8192 | 4039 | 4642 | 6020 |

Test plan (8x MI300X, gfx942, flydsl default backend, no geometry overrides):

```
torchrun --standalone --nproc_per_node={2,4,8} test_op.py                      # also TUNED=1, DTYPE=fp8 TUNED=1, HIDDEN=7168
torchrun --standalone --nproc_per_node={2,4,8} test_ep_backend_parity.py       # also HIDDEN=7168 TOPK=8 EPR=32
torchrun --standalone --nproc_per_node=8 test_graph_capture.py
pytest tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_intranode.py   # 17 passed, 1 skipped (fp4, gfx950-only)
```

Authored with Claude.
@kawhil-amd
kawhil-amd requested review from jhchouuu and kawhil-amd and removed request for jhchouuu October 18, 2026 10:34
Comment thread python/mori/ops/utils.py
# MI350X/MI355X (both 256 CU) are only separable here, not by CU count.
_DID_TO_MODEL = {
0x74A1: "mi300x", # MI300X (gfx942, 304 CU)
0x74A9: "mi300x", # MI300X HF (gfx942, 304 CU)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the diff in the two MI300X version?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the slow reply. They are effectively the same part (gfx942, 304 CU), so they can share the MI300X table. I also mapped the other gfx942 IDs from libdrm's amdgpu.ids that had no model (the VFs and the second MI300X HF ID); the PR description has the list.

detect_model() only knew the base device IDs, so an MI300X VF (0x74b5), the second MI300X HF ID (0x74bd) and an MI325X VF (0x74b9) found no model and, with no gfx942 arch fallback, fell back to the CU-scaled flydsl default that asserts on 304-CU parts. MI308X HF (0x74a8) fell back too and got an untuned geometry. The IDs and names are from libdrm's amdgpu.ids. gfx950 VFs already reach the MI355X table through the arch fallback, so they are left alone.

Test: detect_model() with topology() stubbed returns mi300x for 0x74a1/0x74a9/0x74b5/0x74bd, mi325x for 0x74a5/0x74b9, mi308x for 0x74a2/0x74a8. Not run on VF hardware.

Authored with Claude.
@kawhil-amd

kawhil-amd commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

@jeffdaily Thanks for the contribution, and for including the sweep method and results in the table comment. Having a measured MI300X schedule instead of the CU-scaled fallback is useful. I read the diff only (I have no MI300X to run it on), and I have two comments.

1. The default no longer follows the CU count (tuning_configs.py, _MI300X_DEFAULT / _MI300X_SCHEDULE)

The module docstring says block_num must stay <= CU count. Before this PR, an MI300X went through _cu_scaled_default(), which sizes blocks from cu_count(). Now _MI300X_DEFAULT and every bucket of _MI300X_SCHEDULE use a fixed dispatch_block_num of 128, so the rule is no longer enforced.

The model is picked by PCI DID, and utils.py notes that the CU count shifts with compute partitioning (SPX/DPX/CPX) and CU masking. A partitioned MI300X keeps the same DID but exposes fewer CUs (for example 38 per partition in CPX), so 128 blocks would exceed the CU count there. I have not run a partitioned part, so I do not know whether it hangs or only slows down. Could you check, or add a clamp such as min(block, cu_count()) when the CU count is below the tuned one? If a fixed 128 is intended, a short note in the comment would help.

2. The comment and the first bucket disagree (tuning_configs.py, _MI300X_SCHEDULE)

The header comment says the gather combine wants block 32 / warp 8 "at every count but 65..128 tokens (64x8)". The first bucket is (128, 128, 8, 64, 8), which covers 8..128 tokens, so 8..64 tokens also get combine 64x8, which the comment says is not the best there. Either split the bucket at 64, for example (64, 128, 8, 32, 8) then (128, 128, 8, 64, 8), or fix the comment if 64x8 is also fine below 65 tokens.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants